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Efficacy of the Check & Connect Mentoring Program for At-Risk 
General Education High School Students 


Jessica B. Heppen’, Kristina Zeiser*, Deborah J. Holtzman’, Mindee O’Cummings°*, 
Sandra Christenson”, and Angie Pohl? 


ABSTRACT KEYWORDS 
Although graduation rates are rising, the high school dropout problem dropout prevention 
remains a national crisis, and evidence-based information about efficacy trials 
interventions for at-risk students is critically needed. Prior research _—_ttisk students 
shows that Check & Connect, an individualized mentoring program, — entering 
has positive effects on school persistence and progression for students 

with disabilities. This study examined the efficacy of Check & Connect 

with general education students who showed early warning signs of 

risk for dropping out of high school in a large urban district. The 

sample included 553 students with the lowest predicted probabilities 

of on-time graduation based on attendance, behavior, and course 

performance in Grades 8 and 9. Students were randomly assigned to 

receive a Check & Connect mentor for three years, starting in the 

summer after Grade 9, or not. Findings suggest the program was 

implemented with fidelity, except with students who left district 

schools. Check & Connect did not have any statistically significant 

impacts on measures of engagement, academic progress, the 

likelihood of dropping out, or graduation. These results are discussed 

in the context of other literature on mentoring and dropout 

prevention. 


Although national graduation rates have been ticking upward in recent years, the high 
school dropout problem remains a national crisis. According to the latest statistics from the 
U.S. Department of Education’s National Center for Education Statistics (NCES), 83% of 
public high school students graduate with a diploma within four years (U.S. Department of 
Education, 2016). Thus, nearly one in five students does not successfully graduate from high 
school on time. The problem is particularly pronounced among students of color, students 
who are economically disadvantaged, and students with disabilities (Greene & Winters, 
2005; Stetser & Stillwell, 2014; U.S. Department of Education, 2006). The consequences of 
the dropout problem are serious; recent estimates suggest that, during the next decade, high 
school dropouts will cost the nation about $1.5 trillion in lost wages (Alliance for Excellent 
Education, 2011) and billions more in costs related to public health, crime and justice, and 
public assistance (Levin, Belfield, Muennig, & Rouse, 2007). High school dropouts earn, on 
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average, $7,840 less per year than high school graduates, and their lifetime earnings are 
$1 million less than college graduates (Snyder & Dillow, 2012). 

As of late 2016, the What Works Clearinghouse (WWC) has reviewed the evidence on 25 
“path to graduation” programs, identifying a total of 15 with positive effects on student out- 
comes. One of those programs is Check & Connect, a program originally developed by the 
University of Minnesota’s Institute on Community Integration in collaboration with Minne- 
apolis Public Schools educators to address concerns about low high school completion rates 
among students with disabilities. Check & Connect pairs students with a trained mentor 
who closely monitors their progress in school, matches them with targeted academic and 
social supports, and engages with their families with the goal of increasing student engage- 
ment, performance in school, and school persistence. 

Prior studies, including two randomized controlled trials that met WWC standards with- 
out reservations, demonstrate positive effects of Check & Connect on school persistence and 
progression among students with learning, emotional, or behavioral disabilities (Sinclair, 
Christenson, Evelo, & Hurley, 1998; Sinclair, Christenson, & Thurlow, 2005; What Works 
Clearinghouse, 2015). Although Check & Connect initially focused on students with disabil- 
ities, some school districts now use the program with general education students, and 
research with this broader population is beginning to emerge (e.g., Maynard, Kjellstrand, & 
Thompson, 2014; Guryan et al., 2016). However, prior to the study described in this paper, 
there had been no rigorous test of the efficacy of Check & Connect on high school comple- 
tion for general education students. 

The use of Check & Connect with general education students makes sense theoreti- 
cally, due to research suggesting that the strategies that are component parts of the pro- 
gram can promote school completion and prevent dropout in a broader population. 
Prior research suggests that graduation outcomes may be improved by strengthening 
adult-student relationships (Balfanz & Legters, 2006) and embedding support for stu- 
dents into the regular school schedule (Dynarski & Wood, 1997). Relatedly, strategies 
operationalized by Check & Connect are consistent with several of the recommended 
strategies in the U.S. Department of Education’s Dropout Prevention Practice Guide, 
namely using data to identify at-risk students (Recommendation 1), assigning adult 
advocates to students at risk of dropping out (Recommendation 2), and implementing 
personalized programs to improve students’ classroom behavior and social skills (Recom- 
mendation 4) (Dynarski et al., 2008).’ 

The current study, funded by a grant from the Institute of Education Sciences, tested the 
impact of Check & Connect on student engagement, attendance, academic performance, 
and graduation outcomes among general education students at heightened risk of dropping 
out in a large, urban district. Using student risk indicators measured in Grade 8 and the first 
half of Grade 9, we identified a sample of students at risk of not graduating within four years. 
Within each of 10 participating high schools, we then randomly assigned students in the 
identified sample to participate in Check & Connect or to a business-as-usual control group 
for three years starting in the summer after Grade 9. The study was designed to address the 
following research questions: 


"By linking students with targeted academic supports (e.g., tutoring), Check & Connect mentors also can operationalize 
another practice guide recommendation: Provide academic support and enrichment to improve academic performance 
(Recommendation 3). 
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1. How was Check & Connect implemented in an urban setting with general education 
students identified as being at risk of failing to graduate from high school? 

2. What is the impact of Check & Connect on student outcomes including engagement 
(behavioral, affective, and cognitive), course-taking behavior and academic achieve- 
ment, and likelihood of dropping out and graduating within four or five years? 

In brief, we found that Check & Connect was implemented with fidelity, except with stu- 
dents who left district schools. Contrary to hypotheses and counter to the prior evidence, we 
found the program had no detectable impact on key student outcomes. 

This paper begins by describing the Check & Connect program and then describes the 
identification of the sample of at-risk students, random assignment, and the data sources 
used. The results section addresses the two research questions as well as findings from analy- 
ses of uptake and service contrast, comparing treatment and control students’ reported par- 
ticipation in programs and supports. The final section discusses study implications. 


Description of the Intervention 


Check & Connect is a targeted student intervention designed to encourage student engage- 
ment in school and learning. The program is focused on promoting positive outcomes (e.g., 
academic achievement, credit accrual, and high school graduation) through the provision of 
persistent student support by trained mentors. Check & Connect is designed to serve stu- 
dents in Grades K-12 who are at risk of disengagement or dropout. The program does not 
have strict or prescriptive recommendations for identifying target students for the interven- 
tion, but general guidelines include: (a) determining alterable indicators of student dis- 
engagement in the school or district (e.g., attendance, grades, credits earned, behavior) and 
(b) establishing criteria for the selected indicators of disengagement (e.g., attending less than 
80%-90% of the time, having two Ds or one F, or three or more disciplinary referrals). 

Mentors typically work with students for 11 months a year, for a minimum of two years, 
following mobile youth from school to school within a district. The mentor’s primary goals 
are to promote regular school participation, to keep education a salient issue for students 
and parents, and to fuel motivation to learn through persistent support. In the check compo- 
nent of Check & Connect, mentors systematically utilize school data to monitor students’ 
attendance (e.g., absences, tardiness, skips), social/behavior performance (e.g., suspensions, 
behavioral referrals, detentions), and academic performance (e.g., course grades, credits 
earned). In the connect component, mentors build relationships with students by meeting 
on both a formal (scheduled) and an informal (nonscheduled) basis. During formal meet- 
ings, mentors use check data along with their knowledge of the student to intervene in a 
timely manner. The interventions are intended to address obstacles related to student dis- 
engagement and reestablish and maintain students’ connection to school and learning. Men- 
tors work with school personnel to enhance students’ academic and social competencies; 
mentors also are encouraged to establish connections with students’ families to improve 
home-school communication and promote consistency in messages about the importance 
of education. 

Check & Connect mentors are trained to offer two levels of intervention: basic and inten- 
sive. Basic intervention is provided to all students in the caseload and includes sharing check 
data with students, discussing the relevance of school for students’ goals, fostering opportu- 
nities for participation in school and school activities, and practicing problem-solving 
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strategies to overcome barriers to school success. Students who continue to show high-risk 
behaviors or signs of disengagement receive intensive, individualized interventions. Intensive 
interventions may include arranging for alternatives to suspensions, facilitating participation 
in small-group instruction or tutoring, providing more frequent or intensive problem solv- 
ing (and involving parents and school personnel in intensive problem solving when neces- 
sary), and linking students with community resources (e.g., mental health services, service 
learning opportunities, job programs). Check & Connect does not prescribe specific inten- 
sive interventions; the interventions are at the discretion of mentors and are based on the 
mentors’ understanding of student needs as well as the availability of school- and commu- 
nity-based interventions in the local context. 


Theoretical Framework 


The foundational underpinnings of Check & Connect were drawn from research on resil- 
iency, systems theory, cognitive-behavioral theory, and intrinsic motivation to address the 
complex social problems related to school dropout (see Christenson, Stout, & Pohl, 2012). 
The Check & Connect theory of action centers on student engagement—students’ active par- 
ticipation in school and school-related activities as well as commitment to and investment in 
their learning. Engagement is viewed as a primary mediator for promoting school comple- 
tion (Christenson et al., 2008; Finn, 1989; Grannis, 1994). Engagement is viewed as a multi- 
dimensional construct including behavioral engagement (e.g., attendance, work completion, 
participation in extracurricular activities, disciplinary referrals), affective engagement (e.g., 
sense of belonging, perceived connection with teachers and peers related to learning), and 
cognitive engagement (e.g., perceived relevance of school work, self-regulation toward goals). 

Check & Connect mentors work one-on-one with students to improve these aspects of 
engagement. As the process of reengagement begins, mentors can provide and make avail- 
able targeted and personalized supports and interventions that are aligned to students’ per- 
sonal needs that, in turn, can lead to improved interim academic outcomes including school 
persistence, course performance, credit accrual, and achievement on assessments (e.g., exit 
exams). Improvement in these interim outcomes then leads to a greater likelihood that stu- 
dents will successfully graduate on time. A visual depiction of this theory of action is pre- 
sented in Figure 1. Although the process is shown to be linear, some of the interrelationships 
among components in the theory of action may be cyclical. For example, as students’ interim 
academic outcomes improve (e.g., students receive higher grades), their engagement in 
school (shown as an earlier outcome in the process) may improve, thereby contributing to 
an increased likelihood of further academic success (depicted with a bidirectional arrow). 
(See Christenson et al., 2012, for an expanded theory of action used as part of the Check & 
Connect implementation manual.) 


Method 


This study was a three-year randomized controlled trial designed to test the efficacy of Check 
& Connect on outcomes for general education students identified in Grade 9 as being at risk 
for not graduating on time. The study took place in 10 comprehensive high schools, all 
within a large urban district in California that volunteered to participate. The student popu- 
lation in the study schools averaged 21% White, 14% African American, 48% Hispanic, and 
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Intervention Engagement Outcomes Academic Outcomes 


Check Behavioral Engagement Interim Academic Outcomes Ultimate Academic 

= Mentors systematically = Improved attendance = Improved course Outcomes 
monitor alterable student = Improved behavior at school performance = Decreased dropout 
data (e.g., predictors of "Increased participation in = Improved achievement on rates 
attendance, behavior, and — |_p} extracurricular activities <>) assessments (e.g., exit exams) }—>] = Increased graduation 
academic performance) Affective Engagement = Improved on-track rates rates 

Connect = Increased affective = Increased college preparation = Increased 

= Mentors meet routinely engagement in school (e.g., activities postsecondary 
with students to build sense of belonging) enrollment rates 
relationships with students Cognitive Engagement 


and families, problem-solve 
with students, and diagnose 
student needs 


= Increased cognitive 
engagement in school (e.g., 
self-regulation) 


= Mentors provide timely, 
personalized interventions 
to students 
= Basic intervention (e.g., 
goal setting, 
communicating with 
educators) 


= Intensive intervention 
(e.g., academic support, 
coping strategies, family— 
school support meetings) 


\ t t / 


| Contextual factors (school, community, family) that can either inhibit or facilitate program implementation and attainment of 
outcomes 


Figure 1. Check & Connect abridged theory of action. 


15% Asian (see Table 1). Two thirds (67 percent) of students in these schools were eligible 
for free or reduced-price lunch, and 21% were English language learners. 


Sample Selection 


The goal of this study was to assess the impact of Check & Connect on general education 
students with low probabilities of on-time high school graduation as of the ninth grade. To 
determine students’ probabilities of on-time graduation, we first identified indicators avail- 
able within the district data system that were highly predictive of failure to graduate within 
four years. To determine which indicators predicted failure to graduate, we used data from 
Grades 8 and 9 for cohorts of students who entered the ninth grade in 2005-06 and 2006- 
07, for whom on-time graduation was known. We used logistic regression models to 


Table 1. Characteristics of the 10 study schools, as of 2009-10 school year. 
Characteristics Average percent 


Race/ethnicity 


White 21% 
African American 14% 
Hispanic 48% 
Asian 15% 
Native American <1% 
Eligible for free or reduced-price lunch 67% 
English language learner 21% 
Proficient on English Language Arts State Assessment (2010) 44% 
Proficient on Mathematics State Assessment (2010) 16% 


Source. District-provided school-level data from the 2009-10 academic year. 
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examine the relative predictive strength of several risk factors observed in Grades 8 and 9. In 
the participating study district, we identified the following early warning indicators as the 
risk factors most strongly associated with failure to graduate on time: (a) absent 10% or 
more of enrolled days in Grade 9; (b) had a failing grade in at least one course in Grade 9; 
(c) had a failing grade in Algebra I in Grade 8 or Grade 9; and (d) had at least one “Needs 
Improvement” or “Unsatisfactory” citizenship grade (indicating general behavior and partic- 
ipation in each class) in Grade 9 courses.” 

We then applied the coefficients from this model to the data for students who entered 
Grade 9 in 2010-11 within the 10 participating schools and identified 100 general-education 
students in each of nine of the schools, and 150 in the tenth school, with the lowest predicted 
probability of on-time graduation as of spring 2011 (range 0.34 to 0.94).° Of the 1,050 stu- 
dents identified, a total of 553 agreed to participate and had parental/guardian consent.* We 
then randomly assigned the participating students to condition within school, resulting in 
276 students in the treatment group and 277 students in the control group. The online sup- 
plemental material includes a diagram summarizing the sample selection process 
(Figure S1). 

Treatment students were distributed among five Check & Connect mentors based on the 
school attended in Grade 9. The five mentors who were initially hired and trained for the 
study continued in their roles throughout the full three-year implementation period. Stu- 
dents in the control group were not assigned to a Check & Connect mentor at any time dur- 
ing the study, but they were not prevented from accessing other services available within the 
district. 

Table 2 presents descriptive data on the student sample at baseline, including demo- 
graphics and risk factors measured in Grades 8 and 9. Notably, more than 30% of students 
in the sample were English language learners; approximately 34% were previous English lan- 
guage learners who, prior to Grade 9, met state criteria to be reclassified as fluent English 
proficient; and 72% were Hispanic. The average predicted probability of on-time graduation 
was 0.55. Table 2 demonstrates that the random assignment procedure yielded two groups 
that were not statistically different on any of the baseline measures. 


Data and Measures 


Data collected for this study include implementation data, student administrative records, 
and student survey data. The study team worked with the program providers to collect 
implementation data from monitoring forms that mentors were expected to update weekly 
and submit monthly for each student in their caseload. The monitoring forms included three 
main sections that were used for implementation data: (a) a check data section in which 
mentors recorded student data such as grades, tardies, and suspensions and indicated 
whether the student was considered “high risk” on the basis of the check data and the 


The other possible grades are S (Satisfactory), G (Good), and E (Excellent). 

3To identify at-risk students before the end of the Grade 9 year, we used attendance information from the first three months 
of high school and course grade information through April. 

Recruitment occurred during the summer and frequently involved going to students’ homes. The recruiters did not report 
any active refusals; nonconsent was the result of lack of contact or response. The 553 students in the study did not differ sig- 
nificantly from the 497 nonconsented students in their probabilities of on-time graduation, gender composition, or percent- 
age of students who were African American or Hispanic. Participants were, however, slightly less likely to be White than 
nonconsented students. 
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Table 2. Characteristics of study sample, by condition. 


Observed Adjusted p 

Student characteristic treatment control value 
English language learner status (n = 553) 

Percent English language learner 31.2% 32.0% 0.830 

Percent reclassified English learner 34.8% 34.3% 0.909 

Percent initially fluent in English 34.1% 33.7% 0.923 
Percent female (n = 553) 44.9% 50.9% 0.158 
Percent racial/ethnic group (n = 552) 

Percent African American 12.0% 12.7% 0.800 

Percent White 8.4% 6.7% 0.492 

Percent Asian 5.5% 7.0% 0.476 

Percent Hispanic 72.4% 71.7% 0.866 
Percent students who were absent 10% or more of the enrolled 23.9% 22.4% 0.670 

days in Grade 9 (n = 553) 

Percent students who had at least one Needs Improvement or 90.9% 89.0% 0.469 

Unsatisfactory citizenship (behavior) grade in Grade 9 (n = 553) 

Percent students who failed any course in Grade 9 (n = 553) 98.6% 99.7% 0.197 

Percent students who failed Algebra | (n = 553) 80.1% 79.4% 0.867 

Average predicted probability of graduation (n = 553) 0.55 0.55 0.632 

Average score on Grade 8 English language arts state assessment (n = 319.06 318.23 0.828 

547) 


Source. District administrative records. 

Notes. Differences in student characteristics by condition were tested using a model that accounts for the clustered data struc- 
ture (i.e., students nested within the schools they attended in Grade 9). The treatment group mean is the average value 
observed in the data. The adjusted control group mean was calculated by subtracting the estimated group difference from 
the observed treatment group mean to ensure that comparisons across groups accurately account for differences in student 
background characteristics. 


mentor’s judgment; (b) a student communications section in which mentors recorded the 
number of minutes they met with the student in formal and informal meetings and noted 
the mode of communication (in person, by phone, text, e-mail); and (c) a list of specific 
interventions that were being provided to students. We used the monitoring form data to 
construct four measures of implementation: 

e Fidelity of implementation of the check component. We examined whether mentors 
recorded check data for their students at least once during each month of implementa- 
tion during the academic year (September through June). 

e Frequency of mentor-student interactions. We counted the number of unique days 
that mentors indicated they met (formally or informally) with students and calculated 
the average number of mentor-student interactions per month. 

e Amount of time mentors spent with students. We summed the amount of time men- 
tors reported spending with students in formal and informal interactions during each 
month and calculated the average number of minutes per month. 

e Match between student risk status and provision of basic and intensive interven- 
tions. We examined whether mentors identified students as “high risk” and whether 
they indicated that they provided “basic” or “intensive” intervention. 

The outcome measures for the study were derived from district administrative data 
and student survey data, and are described in Table 3. The district data included infor- 
mation about students’ demographic characteristics and academic background charac- 
teristics (e.g., prior attendance, behavior and course performance grades, and 
achievement test scores), school enrollment in each academic year, attendance, course- 
level transcript data (including both academic and citizenship grades), exit exam pass/ 
fail status, and graduation date and diploma type. Because a number of students left 
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Table 3. Student outcome measures, by outcome domain. 


Measure Description/coding 

Student engagement—Behavioral 
Attendance Indicator of whether a student attended school for 
at least 90% of the days the student was enrolled 
during the 2013-14 school year 

Indicator of whether a student received an 
“unsatisfactory” in two or more courses during 
the 2013-14 school year 

Indicator of whether a student had participated in at 
least one extracurricular activity (e.g., school 
sports, arts or music group, school yearbook, or 
newspaper) since the beginning of summer 2013 


Citizenship grades in 
academic courses* 


Participation in 
extracurricular activities 


Student engagement—Affective and cognitive 

Student Engagement The SEI (Appleton, Christenson, Kim, & Reschly, 
Instrument (SEI) scale 2006) includes 33 survey items used to measure 
five subscales within two domains of student 
engagement: (a) affective engagement, 
including teacher-student relationships 
(Cronbach’s alpha? = 0.89), peer support for 
learning (Cronbach's alpha = 0.88), and family 
support for learning (Cronbach's alpha = 0.83); 
and (b) cognitive engagement, including control 
and relevance of school work (Cronbach's alpha 
= 0.85) and future aspirations and goals 
(Cronbach's alpha = 0.85). 

The SEQ (Steinberg, 1996) includes 12 survey items 
used to measure student engagement in school, 
reflecting perceived relevance of schoolwork, 
teacher-student relationships, teacher 
expectations, and satisfaction with school 
(Cronbach's alpha = 0.78). 


Student Engagement 
Questionnaire Scale 
(SEQ-C) 


Interim academic outcomes—Course performance 

Course failure in fourth Indicator of whether a student failed two or more 

year classes during the 2013-14 school year 

Passed at least one Indicator of whether a student passed at least one 

summer school class summer school class in summer 2013— 
measured among students with district 
administrative data during the second and third 
years of high school so that credit accumulation 
can be measured using the same sample of 
students 


Interim academic outcomes—Achievement 
Passed the high school Indicator of whether a student passed the California 
exit exam High School Exit Exam in both mathematics and 
English by the end of the third year of high 
school 

Number of credits a student earned through the 
beginning of the fourth year of high school 
(including summer 2013)—students in the state 
were required to complete 44 semester credits 
to graduate from high school 


Credit accumulation 


Ultimate academic outcomes—Dropout and graduation 
Dropout Indicator of whether a student had a record of 
dropping out of a high school in the state within 
five years of entering Grade 9 

Indicator of whether a student graduated from a 
school in the state within four years of entering 
Grade 9 


On-time (four-year) 
graduation 
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Source 


District administrative data 
(2013-14)° 


District administrative data 
(2013-14)° 


Student survey (spring 2014) 


Student survey (spring 2014) 


Student survey (spring 2014) 


District administrative data 
(2013-14)° 

District administrative data 
(2011-12 and 2012-13)* 


District administrative data 
(2011-12 and 2012-13)" 


District administrative data 
(2011-12 and 2012-13)" 


District administrative data, 
supplemented with state 
administrative data® 

District administrative data, 
supplemented with state 
administrative data® 


(continued on next page) 


Table 3. (Continued ) 


Measure Description/coding Source 
Five-year graduation Indicator of whether a student graduated from a District administrative data, 
school in the state within five years of entering supplemented with state 
Grade 9 administrative data® 


Note. Two outcomes in the program Theory of Action (Figure 1) that are not included as outcome measures in this paper are 
college preparation activities (interim academic outcome) and postsecondary enrollment (ultimate outcome), which will be 
examined in a subsequent paper. 

*Citizenship grades indicate whether the teacher found the student's behavior and participation in the class to be satisfactory. 
Citizenship grades (Excellent, Good, Satisfactory, Needs Improvement, Unsatisfactory) are independent from course grades, 
and students receive both types of grades for most high school courses (with some exceptions, such as some credit recovery 
courses, physical education, and computer literacy). To identify the at-risk sample at the outset of the study, we used an indi- 
cator of whether students had one or more citizenship grades of Needs Improvement or Unsatisfactory; however, as an out- 
come measure, there was not enough variation in the sample as students moved through high school—nearly all students 
had at least one of these grades per year. Therefore, we raised the criterion to Unsatisfactory in two or more courses. 

The Cronbach's alphas reported in this table are based on the sample of students who participated in the spring 2014 student 
survey. 

‘To have data on these measures, students in the sample had to have been enrolled in schools that submitted transcript data 
to the district data system in the fall and spring of the third year of implementation (2013-14), corresponding to students’ 
fourth year of high school. 

“To have data on these measures, students in the sample had to have been enrolled in schools that submitted transcript data 
to the district data system in the fall and spring of the first and second years of implementation (2011-12 and 2012-13), cor- 
responding to students’ second and third years of high school. 

“All students in the sample had data on these measures. 


the district prior to the end of the study (see the next section on attrition and nonre- 
sponse), we also collected dropout and graduation information from the state longitudi- 
nal data system, allowing us to estimate the impact of Check & Connect on these 
outcomes for the full sample of treatment and control students. The state data system 
includes students who graduated from high school from any public school in the state, 
but it does not allow us to differentiate between nongraduates and students who gradu- 
ated from a high school outside of the state.” 

The study team administered surveys to treatment and control students in the spring of 
2012, 2013, and 2014. These surveys asked students about their participation in social and 
academic activities and contained items from preexisting measures of affective and cognitive 
engagement (see Table 3). To create engagement scale scores, we used partial credit Rasch 
models and standardized the resulting Rasch scores within the analytic samples. We exam- 
ined student reports of having a mentor and involvement in targeted academic supports, 
including tutoring, online credit recovery programs, and college preparatory programs, to 
measure service contrast (ie., the extent to which control students accessed interventions 
similar to those accessed by treatment students). 

For the sake of brevity, only service contrast measures and interim outcomes that 
occurred during the fourth year of high school (as well as credit accumulation and passing 
the high school exit exam by the beginning of the fourth year of high school) are included in 
this paper. These findings demonstrate the impact of Check & Connect on the interim out- 
comes after two full years of implementation. (Earlier interim outcomes are included in the 
online supplemental material.) 


>Seven percent of both the treatment and control groups left the state permanently at some point following the summer of 
2011 (without first having graduated). 
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Weighting for Attrition and Nonresponse 


Our main impact analyses estimate intent-to-treat effects, meaning the analyses assess the 
impact of the program on outcomes for all students who were randomly assigned, not just 
for students who actually participated in the program. However, many of the interim student 
outcomes were derived from district administrative data, and approximately half of the study 
students did not remain in the data system throughout the full duration of the study. In 
addition to moving out of the district and dropping out of high school, many students trans- 
ferred to local charter schools that did not submit detailed transcript data to the district. 
Figure 2 shows the percentages of treatment and control students in different school settings 
across the three years of program implementation. Although these percentages were similar 
by condition, attrition bias in the analysis of some of the study outcome measures is possible. 

In particular, interim outcomes (see Table 3) based on multiple years of administrative 
data were subject to possible attrition bias. We measured students’ credit accumulation, per- 
formance in summer school, and completion of the high school exit exam by the beginning 
of students’ fourth year of high school. To have data on these measures, students in the sam- 
ple had to have been enrolled in schools that submitted course-level transcript data to the 
district data system during the first two years of implementation (2011-12 and 2012-13), 
corresponding to students’ second and third years of high school; 40.2% of treatment stu- 
dents and 40.4% of control students were missing administrative data during at least one of 
these years. We examined course failures, attendance rates, and citizenship grades during 
students’ fourth year of high school. To have data on these measures, students in the sample 
had to have been enrolled in schools that submitted course-level transcript data to the dis- 
trict data system in the fall and spring of students’ fourth year (2013-14, the third year of 
implementation); 46.4% of treatment students and 47.3% of control students were missing 
administrative data for at least one semester of the year. Interim outcomes based on survey 
data, however, were less subject to attrition bias because data collectors followed up with 


100% 
90% 
» 
sill Snot enrolled in district 
70% 
60% ® enrolled in non-participating district 
50% school not providing transcript data 
40% B enrolled in non-participating district 
school providing transcript data 
30% ¢ 
® enrolled in one of the 10 
20% participating schools 
10% 
0% 


treatment cones | en ois trent] paren 


2011-12 2012-13 2013-14 


Figure 2. Percentage of treatment and control students in different settings during each year of imple- 
mentation. The “enrolled in non-participating district school not providing transcript data” category 
includes district-authorized charter schools that are not required to submit transcript data to the district. 
The “not enrolled in district” category includes students who transferred outside of the district and stu- 
dents who dropped out of high school. Source. District administrative records. 
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students who had left district schools. Rates of missing data for interim outcomes measured 
by the spring 2014 student survey were 19.9% for treatment students and 19.5% for control 
students; thus about 80% of both groups were included in the analysis of engagement out- 
comes and service contrast measures. Figure $2 in the online supplemental material provides 
the numbers of treatment and control students in the analytic sample for each outcome. 
Although overall attrition rates were high in this study of at-risk, highly mobile students, dif- 
ferences in attrition rates between treatment and control students (i.e., differential attrition) 
were below 2 percentage points at each period of follow-up.° 

We excluded students with missing outcome data from analyses of the interim outcomes 
but took attrition and nonresponse into account by applying inverse probability weights 
(IPWs; see Wooldridge, 2007). We applied the IPWs under the assumption that outcome 
data are not missing completely at random but that outcome data are missing at random 
once we account for observed student characteristics such as prior achievement and demo- 
graphic information. Assuming that the observed student characteristics in the IPW model 
accurately account for students’ probabilities of having nonmissing outcome data, applying 
these weights allows us to generalize our findings to the original sample. However, while 
IPWs are calculated based on observed demographic and achievement characteristics, IPWs 
cannot account for differences in unobservable characteristics such as student dispositions. 
Even after applying IPWs, the treatment and control students in the analysis samples for the 
interim outcomes may differ on these unobservable characteristics. 

We estimated weights by first predicting students’ probabilities of having nonmissing out- 
come data based on observed student demographic and academic background characteristics 
in Grades 8 and 9. To estimate predicted probabilities, we used a generalized boosted logistic 
regression technique implemented in the twang package in R (Ridgeway et al., 2013). Proba- 
bilities of having nonmissing data were estimated separately for treatment and control stu- 
dents. We used the inverse of these probabilities as weights in our analyses. We calculated 
three separate IPWs to account for: (a) students with missing administrative data during the 
third year of implementation; (b) students with missing administrative data for at least one 
semester during the first two years of implementation; and (c) students who did not respond 
to the spring 2014 survey. Thus, students with higher probabilities of having missing data 
(ie., lower probabilities of having nonmissing data) are given more weight in analyses 
because their characteristics are more similar to the characteristics of students who were not 
included in the sample due to attrition (transferring to a school that did not submit tran- 
script data to the district data system, transferring out of the district, or dropping out of high 
school) and nonresponse. 

To assess the performance of the attrition and nonresponse weights, we examined the 
characteristics of all students in the original sample, the unweighted background characteris- 
tics of our analytic samples (i-e., students with nonmissing outcome data), and the weighted 
background characteristics of our analytic samples to observe the extent to which IPWs 
reduced differences between the unweighted analytic sample and the original sample. In gen- 
eral, we observed that higher risk students were more likely to have missing data due to 
mobility or survey nonresponse. Students in the unweighted analytic samples tended to have 


°For both treatment and control groups, the primary source of attrition was transfer. Only six treatment students and eight 
control students dropped out directly from district schools in their second, third, or fourth year of high school; most trans- 
ferred out of the district first (e.g., to a charter school) or did not drop out until after their fourth year. 
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higher prior achievement test scores and probabilities of on-time high school graduation, 
were less likely to be English language learners, and were less likely to exhibit individual aca- 
demic risk indicators than students in the original sample. After applying IPWs, the 
weighted student background characteristics of the analytic samples were more similar to 
the observable characteristics of the original sample. (See Table S1 in the online supplemen- 
tal material.) Nevertheless, analyses of interim outcomes based on district data may be con- 
sidered as nonexperimental due to the substantial amount of missing data associated with 
administrative records, and the possibility that the IPWs do not account for differences in 
unmeasured characteristics between treatment and control students who remained in district 
schools. Analyses of graduation and dropout outcomes were based on the full original sam- 
ple of students randomly assigned to condition. 


Analytic Methods 


Following an intent-to-treat (ITT) approach, the main impact analyses compared outcomes 
for students assigned to the treatment group to outcomes for students assigned to the control 
group after controlling for student demographic and academic background characteristics 
that were measured prior to treatment assignment. The main impact analyses estimated 
weighted multiple regression models as follows: 


x 
Yj =a + Bol + So BXxi + &; (1) 
1 


where Y; represents an outcome for student i, T; is the treatment indicator (0 for control 
and 1 for treatment), By is the estimated program effect, and X,; is a vector of pre-ran- 
dom-assignment background characteristics whose accompanying regression coefficients 
are B,. X,; includes student race and ethnicity, gender, predicted probability of on-time 
graduation (based on attendance and course performance in Grades 8 and 9 and used 
for sample selection), and a vector of school fixed effects identifying the schools from 
which the participating students were sampled. These school fixed effects indicators 
remain unchanged even if students moved to a different school during the study. Finally, 
€; is a random error term. When Y; is binary (e.g., on-time graduation), we estimate 
Equation 1 with a logit link function. 

To supplement the ITT analyses, we used an instrumental-variables approach to estimate 
the impact of Check & Connect for treatment group students who actually received the treat- 
ment (i.e., a treatment-on-the-treated or TOT analysis). A full description of these analyses 
can be found in the online supplemental material. 


Results 


This section begins by presenting results for analyses related to implementation of Check & 
Connect. It then examines overall levels of uptake among treatment students as well as ser- 
vice contrast to consider whether program impacts could be attenuated due to low uptake or 
control students’ participation in activities similar to those of Check & Connect. The section 
concludes by presenting results on the impact of the program on the outcomes listed in 
Table 3. 
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Implementation of Check & Connect 


Our findings regarding how the program was implemented in the study follow a description 
of the mentor hiring and training processes, which were crucial to implementation. 


Mentor Hiring and Training 

Mentors were hired by the district and trained for their role by the program developers from 
the University of Minnesota’s Institute on Community Integration. The district first identi- 
fied a coordinator, who was the district’s existing Dropout Prevention Program Manager. 
The coordinator and other district staff and leaders conducted a search for the mentors in 
spring 2011, and the pool of applicants was narrowed based on input from the program 
developers. Five mentors were hired as district employees in summer 2011. They all had pre- 
vious experience working with at-risk youth (e.g., as a counselor or parole officer), three were 
bilingual, and at the time of hire, all had at least a bachelor’s degree and two had a master’s 
degree in social work or counseling. 

Prior to implementation, the mentors received a Check & Connect manual and attended a 
two-day training session provided by the program developers in summer 2011. Topics of the 
training included the context of the dropout problem, evidence of prior effectiveness of 
Check & Connect, the role of the mentor, the criteria mentors should use to classify students 
as needing basic or intensive intervention based on the check data, and types of interventions 
that mentors could provide (see Christenson et al., 2012, for more information about mentor 
training). Each year of the study, the program developers provided ongoing professional 
development to the coordinator and the five mentors on strategies to enhance students’ 
intrinsic motivation and engagement and challenges raised by the mentors. In addition, the 
program developers held biweekly phone calls with the coordinator to provide ongoing sup- 
port for implementation, and the coordinator held weekly in-person meetings with the 
mentors.’ 

For the study, the five mentors were each assigned a caseload of between 50 and 60 stu- 
dents.* Each mentor’s caseload was spread across 2 of the 10 high schools that agreed to par- 
ticipate in the study at the outset.” The five mentors remained in their roles throughout the 
three years of implementation (with the exception of two maternity leaves, which were cov- 
ered by substitute mentors identified by the district). 


Implementation Findings 
The expectation for mentors was to check student progress weekly using the monitor- 
ing form and to meet with students at least twice per month (or more frequently if 


’The coordinator also met one-on-one with mentors on an as-needed basis if mentors faced issues serving the students or if 
the coordinator’s review of the monitoring forms suggested that additional meetings were necessary. 

8 According to the Check & Connect manual, the assumption is that each mentor will spend approximately 0.8 hours per stu- 
dent each week. Because the mentors in this study were employed full time, program developers determined that mentors 
could manage a caseload of 50 students (40 hours/0.8 hours per student). Although this might seem high by the standards 
of well-known mentoring programs such as Big Brothers Big Sisters of America (Tierney, Grossman, & Resch, 1995), it is low 
compared to typical caseloads for high school guidance counselors, which nationally average above 300 (College Board, 
2012). 

°To limit the number of schools to which mentors needed to travel to meet with students, mentors were matched with stu- 
dents based on the high school students attended in the ninth grade. However, the coordinator carefully considered mentor 
and student characteristics in placing the mentors; all of them lived in the communities they served and had ethnic back- 
grounds resembling those of many of the students they served. Approximately 50% of students were matched to a mentor 
of the same gender and 58% were matched to a mentor of the same race/ethnicity. 
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determined necessary through check data) beginning in fall 2011. Initially, mentors also 
were expected to maintain relationships with students who remained in the general 
geographic area of the district even if the students enrolled in a charter school or were 
no longer enrolled in school. However, it was not possible, in most cases, for mentors 
to obtain check data for students who transferred to charter schools that did not sub- 
mit transcript data to the district data system or left the district (see Figure 2 for 
percentages). 

In the following subsections, we examine the extent to which mentors conducted the 
check component of the program and the frequency of mentor-student interactions as 
measures of fidelity. We also examine, for descriptive purposes, the amount of time treat- 
ment students spent in formal and informal mentor-student meetings during the three years 
of implementation. Because of the high rates of mobility among study participants, we pres- 
ent implementation findings (a) for the full sample of treatment students and (b) separately 
for nontransfer students (defined as students who were enrolled in the 10 participating 
schools as well as in other district schools that submitted course-level transcript data to the 
district data system) and transfer students (defined as students who were enrolled outside of 
the district or in district-authorized charter schools that did not submit student-level tran- 
script data to the district data system, and students who were not enrolled in school).'® For 
descriptive purposes, we also examine whether mentors’ provision of basic and intensive 
services aligned with students’ at-risk status." 


Check Data. Mentors were expected to record students’ check data related to attendance, 
course failures, and behavioral issues weekly on their monitoring forms and then submit 
these forms to the coordinator monthly. The coordinator checked the submitted forms to 
identify issues for discussion in her weekly meetings with the mentors. We examined 
whether mentors recorded these data at least once every month during each academic year 
as a measure of fidelity of implementation of the check component of the program. As 
shown in Table 4, during the first school year after random assignment, mentors recorded 
check data each month for about 65% of the treatment students. In the second and third 
years of implementation, mentors recorded check data on monthly monitoring forms for 
successively smaller percentages of students.'? As expected, the percentage of students with 
monthly check data was consistently higher for the nontransfer treatment students (i.e., 
those who maintained enrollment in a school that submitted student-level transcript data to 
the district data system during the academic year). However, only 60% of nontransfer treat- 
ment students had monthly check data during the third academic year of program 


1° Any student who was not enrolled in a transcript-data-providing school in the fall or spring of the academic year was classi- 
fied as a transfer student within that school year. 

"Another important aspect of implementation is the quality of the student-mentor relationship. In the spring 2014 survey, 
73% of the 190 treatment students who responded to survey items related to student-mentor relationship quality agreed 
or strongly agreed with all 11 statements describing positive aspects of the relationship (e.g., “My mentor respects me,” “My 
mentor really cares about me,” “I could ask my mentor for help if | had a problem”). Detailed findings about relationship 
quality are not included in this article due to the limited variation in students’ responses. 

' Several factors may have prevented mentors from recording student data on at least a monthly basis, such as students mov- 
ing to a school that did not submit transcript data to the district data system and technical difficulties accessing a new online 
data system that was initiated during the study. Therefore, we also examined the percentage of treatment students for 
whom check data were recorded on monitoring forms at least once in the fall and once in the spring of the academic year. 
Using this more lenient threshold, the percentage of students who experienced the check component of the program with 
fidelity was higher: 92%, 79%, and 63% during the first, second, and third year of implementation, respectively. 
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Table 4. Percentage of treatment students whose mentors recorded check data on each submitted moni- 
toring form, by year. 


All students Nontransfer students Transfer students 
Percent of Percent of Percent of 
Academic Number of students with Number of students with Number of students with 
year students monthly check data students monthly check data students — monthly check data 
2011-12 276 64.9% 226 73.9% 50 24.0% 
2012-13 276 45.7% 169 71.6% 107 4.7% 
2013-14 276 32.6% 148 60.1% 128 0.8% 


Source. Monthly mentor monitoring forms. 

Notes. Academic year is defined as September through June. Students with missing monitoring forms during the academic 
year were classified as having monthly check data, or not, based on whether check data were recorded on all submitted 
monitoring forms. Students for whom no monitoring forms were submitted within the academic year were classified as not 
having check data. 


implementation. It is possible that mentors monitored students’ academic progress without 
recording the check data onto the form. 


Frequency of Mentor-Student Interactions. The monitoring form data indicate that, on 
average, mentors interacted with the students in their caseload about as frequently as 
recommended for faithful implementation of the program (twice per month) during the 
academic year, with fewer meetings during the summer months. Frequency of interac- 
tions differed by students’ transfer status. On average, nontransfer students interacted 
with their mentors twice per month during the first academic year of implementation 
and three times per month during the second and third academic years. In addition, 
nontransfer students averaged 0.8 interactions per month during the first summer and 
1.9 interactions per month during the second summer (see Table 5). Mentors met less 
frequently with transfer students, averaging one or fewer meetings per month during 
each academic year and summer. This lower frequency would be expected, particularly 
in the second and third years, when the program developer and implementation team 
revised the expectation for frequency of meetings with transfer students to once per 
month, and not necessarily in person, due to the time required for traveling to meet 
with students who had transferred to other schools. 


Table 5. Average number of meetings per month for mentors and their students between Septem- 
ber 2011 and June 2014. 


All Students Nontransfer Students Transfer Students 
Average Average Average 
Number number of Number number of Number number of 

of meetings Standard of meetings Standard of meetings Standard 
Time period students permonth deviation students permonth deviation students permonth deviation 
AY 2011-12 276 2.0 0.9 226 2.2 0.8 50 del 0.9 
Summer 2012 276 0.7 0.8 226 0.8 0.9 50 0.4 0.5 
AY 2012-13 276 2.2 13 169 3.0 0.9 107 0.9 09 
Summer 2013 276 1.3 1.4 169 1.9 1.4 107 0.4 0.7 
AY 2013-14 276 2.0 1.6 148 32 i 128 0.7 1.0 


Source. Monthly mentor monitoring forms. 
Notes. Academic year (AY) is defined as September through June; summer as July and August. 
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Table 6. Average number of minutes per month in formal and informal mentor—student meetings 
between September 2011 and June 2014. 


All students Nontransfer students Transfer students 
Average Average Average 
Number number of Number number of Number number of 
of minutes per Standard of minutes per Standard of minutes per Standard 


Time period _—_ students month — deviation students month deviation students month deviation 


AY 2011-12 276 34.6 15.0 226 36.8 12.8 50 24.4 19.3 
Summer 2012 276 14.3 22.4 226 15.8 23.8 50 74 12.6 
AY 2012-13 276 37.9 21.8 169 49.5 14.8 107 19.5 18.0 
Summer 2013 276 16.9 17.9 169 23.4 17.9 107 6.7 12.2 
AY 2013-14 276 39.3 32.5 148 60.9 24.4 128 14.2 20.6 


Source. Monthly mentor monitoring forms. 
Notes. Academic year (AY) is defined as September through June; summer as July and August. 


Amount of Mentor-Student Meeting Time. Table 6 shows that, among nontransfer stu- 
dents, formal and informal meeting time averaged about 37 minutes per month during the 
first academic year of implementation, 50 minutes per month during the second year, and 
61 minutes per month during the third year (plus, on average, 16 minutes per month during 
the first summer and 23 minutes per month during the second summer). The average num- 
ber of minutes per month mentors spent with transfer students was lower during all time 
periods. 

Not shown in Table 6, on average, across the three academic years and two summers of 
implementation, students in the treatment sample met with their mentor for a total of 
1,180 minutes, or about 20 hours. Among the 136 students who maintained enrollment in 
data-providing schools during this time frame, total meeting time averaged 1,565 minutes, 
or 26 hours per student over the full three years. 

The data on amount of meeting time are presented as descriptive implementation analy- 
ses, not fidelity analyses, because the Check & Connect program is not prescriptive about 
how much time mentors should spend with each student. Rather, mentors are encouraged 
to use their judgment in allocating time across their caseload, based on their sense of stu- 
dents’ individual needs and circumstances. 


Student Risk Status and Basic Versus Intensive Interventions. The monitoring form asked 
mentors to indicate, based on the check data and their own judgment, whether students 
were at “high risk.” Mentors could provide or recommend intensive interventions based 
on their judgment, and they were trained to factor students’ risk designation into inter- 
vention decisions. We used the monitoring forms to examine the alignment between 
mentors’ designation of students’ high-risk status and the level of intervention they pro- 
vided or recommended. As shown in Table 7, mentors provided intensive services to a 
majority of the students they identified as high risk (66% of treatment students in Year 
1, 85% in Year 2, and 80% in Year 3), but they also provided intensive intervention to a 
majority of their students who were not designated as high risk (53% in Year 1, 68% in 
Year 2, and 70% in Year 3). Provision of intensive intervention to lower risk students 
may have been appropriate given that mentors were instructed to tailor the intervention 
to students based on student needs that may not have been captured solely in recorded 
check data. 
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Table 7. Percentage of monitoring forms indicating provision of intensive services, by mentor-designated 
student risk status between September 2011 and June 2014. 


Identified by mentors as high risk Identified by mentors as not high risk 
Academic Total number of Number of Percentage receiving Number of Percentage receiving 
year monitoring forms — monitoring forms intensive services monitoring forms intensive services 
2011-12 2,723 1,610 66% 1,113 53% 
2012-13 2,831 2,132 85% 699 68% 
2013-14 2,210 1,707 80% 503 70% 


Source. Monthly mentor monitoring forms. 


Summary of Implementation Findings. Overall, Check & Connect was implemented with 
fidelity for treatment students who remained in district schools, except for documentation of 
the check component. On average, we observed increases over time in both the frequency of 
mentor-student interactions and the amount of meeting time among nontransfer students, 
suggesting improvement in mentoring relationships throughout the study. Findings also 
indicate that, although mentors attempted to contact transfer students and obtain informa- 
tion about their academic behaviors, enrollment in a nondistrict school or movement out- 
side of the district created structural barriers that substantially lowered the level of 
implementation for these students. Finally, mentors largely provided intensive interventions 
to most of their students regardless of whether students were identified as high risk. 


Overall Program Uptake and Service Contrast 


To set a context for the impact findings, it is important to understand the extent to which the 
students in the treatment group received the “full” treatment (that is, both the “check” and 
the “connect” components) during the three-year study. It is also important to examine the 
service contrast, that is, the extent to which the experiences of control students may have 
resembled or differed from the experiences of treatment students. 


Rates of Treatment Receipt 

We defined “receiving the treatment” as (a) meeting with a mentor at least 20 times during 
the year and (b) having check data on all monitoring forms during the school year. Approxi- 
mately 59% of students in our treatment group sample received the treatment for at least 
one year, 34% received the treatment for at least two years, and 14% received the treatment 
for all three years of implementation. Paradoxically, the 14% of students who received Check 
& Connect for all three years may have been the students least in need of it; their average 
predicted probability of on-time graduation was 0.66, compared with 0.53 for the rest of the 
treatment group students. This speaks to the difficulty of fully implementing Check & Con- 
nect with a population for whom high risk and high mobility are closely intertwined.”* 


'3if we relax the “check” requirement to having check data at least once in the fall and once in the spring, 75% of the treat- 
ment group received the treatment for at least one year, 59% of the treatment group received the treatment for at least 
two years, and 34% of the treatment group received the treatment for all three years of implementation. Even with this 
more lenient definition, students who received the treatment for two or three years still had higher average predicted prob- 
abilities of on-time graduation than did the remaining treatment students. 
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Table 8. Comparison of treatment and control students’ participation in targeted academic supports, 
spring 2014. 


Treatment Control Estimated 
Outcome observed mean adjusted mean difference p value N 
Reported having a mentor 88.4% 32.6% 55.8% 0.000 443 
Reported participating in 
targeted academic supports 
Tutoring 13.8% 17.2% —3.4% 0.352 425 
Online credit recovery 47.1% 48.5% —14% 0.787 426 


(e.g., AVID) 16.2% 17.4% —1.2% 0.741 426 


Source. Spring 2014 student survey. 

Notes. IPWs are applied to all analyses to account for missing outcome data. The treatment group mean is the average value 
observed in the data. The adjusted control group mean was calculated by subtracting the estimated group difference from 
the observed treatment group mean to ensure that comparisons across groups accurately account for differences in student 
background characteristics. 


Service Contrast 
In this study, treatment students were assigned a Check & Connect mentor, but control stu- 
dents were not prevented from meeting with mentors or from receiving services that were 
not part of the Check & Connect program. On the spring survey, treatment and control stu- 
dents were asked to report whether they had someone in their life they considered to be a 
mentor to them (however, the survey did not provide a definition of mentor). In spring 
2014, after the third year of the intervention, 89% of treatment students reported having a 
mentor compared with 33% of control students (see Table 8), indicating that treatment stu- 
dents were, in fact, significantly more likely than control students to report having a mentor. 
One way in which Check & Connect mentors could influence student outcomes is by 
facilitating students’ access to targeted academic supports available within the school. Treat- 
ment students should theoretically have been more likely than control students to participate 
in targeted academic support activities due to mentors’ systematic monitoring of academic 
progress, holding regular meetings with their students, and connecting students with these 
types of supports and resources. However, control students also may have had access to these 
same supports. To compare participation in targeted academic supports by condition, we 
examined students’ self-reported rates of participation in tutoring, online credit recovery, 
and college preparatory programs such as AVID and GEAR UP. As shown in Table 8, treat- 
ment and control students did not differ in their rates of participation in these activities. 
Given that access to targeted academic support is a key mechanism through which Check & 
Connect should have an impact on student outcomes, control students’ participation in these 
types of academic supports could attenuate the impact of Check & Connect. 


Program Impacts 


The goal of the Check & Connect program is to improve students’ engagement in school so 
that they are more likely to make academic progress and persist in high school. The pro- 
gram’s theory of action assumes that students’ academic outcomes can be improved by 
improving various aspects of their engagement with school. 

Panel A of Table 9 shows the results of analyses testing the impact of Check & Connect on 
the behavioral engagement measures: attendance, citizenship (behavior and participation) 
grades, and participation in extracurricular activities. Counter to our hypotheses, Check & 
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Table 9. Comparison of treatment and control students’ outcomes. 


Treatment Control Estimated 
Outcome observed mean adjusted mean difference pvalue N 


Panel A: Behavioral engagement (participation), spring 2014 


Attended school for at least 90% of days enrolled 81.6% 73.1% 8.6% 0.120 292 

Had fewer than two Unsatisfactory citizenship 70.1% 68.3% 1.8% 0.767 293 

grades 

Participated in at least one extracurricular activity 53.3% 50.7% 2.6% 0.617. 421 
Panel B: Affective engagement, spring 2014 

(SEI) Teacher—-student relationships —0.04 0.02 —0.06 0.562 429 

(SEl) Peer support for learning —0.05 0.02 —0.07 0.452 427 

(SEl) Family support for learning 0.00 0.00 0.00 0.982 428 
Panel C: Cognitive engagement, spring 2014 

(SEI) Future aspirations and goals —0.07 0.03 —0.10 0.327 428 

(SEI) Control and relevance of school work —0.04 0.03 —0.06 0.509 429 

Student engagement scale (SEQ-C) —0.06 0.04 —0.10 0.269 429 
Panel D: Course performance and academic achievement 

Successful completion of at least one summer course 57.4% 33.6% 23.8% 0.000 329 

Fewer than two course failures in fourth year 65.7% 69.2% —3.5% 0.564 290 

Ever passed high school exit exam 73.3% 75.9% —2.6% 0.666 329 

Credit accumulation 34.4 34.1 0.2 0.801 329 
Panel E: Dropout, on-time, and five-year graduation 

Dropped out of a high school within the state 22.1% 18.9% 3.2% 0.400 552 

On-time graduation within the state 52.2% 52.6% —0.5% 0.921 552 

Five-year graduation within the state 59.4% 58.0% 14% 0.762 552 


Sources. District administrative records (first and second rows of Panel A; all rows of Panel D); spring 2014 student survey (third 
row of Panel A; all rows of Panels B and C); district and state administrative records (all rows of Panel E). 

Notes for all panels. |PWs are applied to all analyses except in Panel E to account for missing outcome data. The treatment 
group mean is the average value observed in the data. The adjusted control group mean was calculated by subtracting the 
estimated group difference from the observed treatment group mean to ensure that comparisons across groups accurately 
account for differences in student background characteristics. 

Panel A notes. Attendance and citizenship grades are measured among all students who were in district transcript data for 
both semesters of the 2013-14 academic year. 

Panels B and C notes. Differences between treatment and control students are presented as effect sizes. 

Panel D notes. Course failures are measured among all students who were in district transcript data for both semesters of the 
2013-14 academic year. Summer course completion, passing of the exit examination, and credit accumulation are measured 
among all students who were in district transcript data for both semesters of the 2011-12 and 2012-13 academic years. 


Connect did not have an impact on any of these measures of behavioral engagement. The 
effect on student attendance was notable in magnitude (and in the hypothesized direction), 
but not statistically significant—82% of treatment students, versus 73% of control students, 
attended at least 90% of enrolled days during the 2013-14 academic year (p = 0.12). These 
results should be interpreted with caution because, as noted earlier, inclusion in the analysis 
sample for attendance and citizenship grades required that students were enrolled in a dis- 
trict school during the third year of implementation. Therefore, many students in the origi- 
nal random assignment sample were excluded due to missing outcome data, and while 
IPWs were calculated based on observed student characteristics, it is possible that the treat- 
ment students and control students included in the analysis of these outcomes differed in 
unobservable ways. 

Also contrary to our hypotheses, Check & Connect did not have an impact on students’ 
affective or cognitive engagement with school (see Table 9, Panels B and C). After three years 
of program implementation, treatment and control students did not significantly differ on 
scales measuring teacher-student relationships, peer support for learning, family support for 
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learning, future aspirations and goals, or control and relevance of school work. Check & 
Connect also had no statistically significant impact on students’ SEQ-C scores, another mea- 
sure of affective and cognitive engagement. 

To examine the impact of Check & Connect on academic outcomes, we focused on out- 
comes observed immediately before and during students’ fourth year of high school. First, 
because we observed that a majority of students in our study were credit deficient in earlier 
academic years, we examined students’ likelihood of passing courses during the summer 
before the fourth year of high school as an interim outcome measure related to course per- 
formance. As shown in Panel D of Table 9, treatment students were significantly more likely 
than control students to pass at least one summer course in the summer prior to their fourth 
year of high school.'* However, we did not observe statistically significant differences 
between treatment and control students in the number of credits students accumulated from 
the start of high school through the summer of 2013.'” Check & Connect also had no signifi- 
cant impact on students’ likelihood of failing fewer than two courses during the fourth year 
of high school or passing the state high school exit exams in mathematics and English lan- 
guage arts. Again, applying IPWs to statistical models attempts to address the fact that stu- 
dents who were not enrolled in transcript-data-submitting schools during the first two years 
of the study (for the credit accumulation and exit exam outcomes) or the third year of the 
study (for the course failures outcome) had missing outcome data and therefore were not 
included in analyses. Nevertheless, these results should be interpreted with caution because 
the treatment and control students who were included may have differed in unobservable 
ways for which IPWs could not account. 

Table 9, Panel E displays results testing the impact of Check & Connect on students’ like- 
lihood of dropping out of high school and their likelihood of graduating within four or five 
years. Unlike the interim outcomes, these results are based on data for the full original exper- 
imental groups of treatment and control students regardless of whether they remained 
enrolled in district schools throughout the study. We found that the likelihood of dropping 
out of high school within the state did not significantly differ for treatment and control stu- 
dents (22% versus 19%). Likewise, there were no statistically significant differences between 
treatment and control students’ likelihood of graduating. Fifty-two percent of treatment stu- 
dents and 53% of control students graduated within four years of high school entry. Fifty- 
nine percent of treatment students and 58% of control students graduated within five years. 

In summary, Check & Connect did not increase students’ likelihood of graduating high 
school (or decrease their likelihood of dropping out), as hypothesized, perhaps because the 
program did not have a measurable impact on key interim outcomes, including student 
engagement and academic performance.'° On average, students in both the treatment and 
control groups continued to be academically at risk throughout the study. The one possible 


“Treatment students also were more likely than control students to take courses that summer before their fourth year of high 
school (60.1% versus 36.8%, p < 0.001). 

‘We also conducted alternative unweighted analyses using the full sample of treatment and control students that compared 
the number of credits students accumulated in the school district by the end of the 2012-13 school year regardless of 
whether students left the school district (or transferred to a school that did not submit transcript data to the district data sys- 
tem). Although these analyses showed a lower average number of credits accumulated during the first three years of high 
school (due to the inclusion of transfer students in the analytic sample), differences between treatment and control students 
were not statistically significant. 

'® Analyses of engagement data and course performance data from the first two years of implementation were similar, indicat- 
ing no significant program impact (see the online supplemental material). 
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sign of program impact was the successful completion of credit recovery courses during 
summer 2013, the summer before students’ fourth year of high school, but this did not trans- 
late into an increased likelihood of graduating within four or five years after high school 


entry. 


Treatment-on-the-Treated Effects 


We also performed TOT analyses that investigated the effect of treatment among students 
who received the treatment. As noted in the section above on program uptake, our definition 
of “receiving the treatment” was based on a combination of the “check” and “connect” com- 
ponents of the program (specifically, meeting with mentors at least 20 times during the year 
and having check data recorded on every submitted monitoring form during the school 
year). Similar to the ITT results, the only outcome showing a significant impact was success- 
ful completion of summer school courses; this result was significant regardless of the number 
of years (one, two, or three) students had received the treatment. The TOT analysis revealed 
no measurable impact on any of the other outcomes for treatment students who received the 
treatment during at least one, at least two, or all three years of implementation. We observed 
these same results with an alternative, relaxed definition of receiving the treatment (meeting 
with mentors at least 20 times during the year and having check data on monitoring forms 
once during the fall and once during the spring). Further detail is available in the online 
appendix. 


Discussion 


Although matching children and young adults with adult mentors has long been a popular 
method of youth development, the efficacy of mentoring has only been seriously studied 
within the last 25 years (Blakeslee & Keller, 2012). From the first major experimental study 
of the Big Brothers Big Sisters of America program (Tierney et al., 1995), a robust and grow- 
ing body of evidence suggests that mentoring programs can yield positive outcomes to youth 
in such areas as academics, behavior, attitudes and motivation, and physical health (Biggs, 
Musewe, & Harvey, 2014; DuBois, Portillo, Rhodes, Silverthorn, & Valentine, 2011). How- 
ever, the impact of mentoring on these outcomes tends to be modest in magnitude—typi- 
cally about two tenths of a standard deviation (DuBois et al., 2011), and can sometimes be 
described as a slowing of negative trajectories rather than outright improvements (Rhodes & 
DuBois, 2008). In addition, not every mentoring program shows a clear improvement in out- 
comes, and few studies of the effects of mentoring have examined educational attainment as 
an outcome (DuBois et al., 2011). 

The current study was designed to address a gap in the literature regarding the impact of 
Check & Connect on general education students not eligible for special education services 
who displayed early warning risk signs in their first year of high school. Contrary to previ- 
ously published experimental studies demonstrating a positive impact on school persistence 
and progression (Sinclair et al., 1998; Sinclair et al., 2005), this study found that Check & 
Connect had no detectable impact on the key study outcomes during or following three years 
of implementation. These outcomes include measures of engagement (attendance, behavior, 
participation in school activities, and affective and cognitive engagement with school) and 
academics (course grades and passing exit exams required for graduation). On all of these 
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measures, treatment students’ outcomes were not significantly different from control stu- 
dents’ outcomes. Most notably, Check & Connect did not have a measurable impact on stu- 
dents’ likelihood of dropping out of or graduating within four or five years of entering high 
school. 

As noted in the analysis and results sections, some of the outcomes reported in this paper 
were based on treatment-control group comparisons that differed from the original ran- 
domly assigned groups due to missing outcome data. The interim outcomes based on district 
records required that students were enrolled in a given year (or across years) for inclusion in 
the analysis; students who were not enrolled at the relevant times had missing data. We 
observed that mobility rates were similar by condition, despite the fact that one might expect 
Check & Connect to help students persist in district schools. It is possible that the treatment 
and control students who were included in analyses of interim outcomes based on district 
records differed in unobservable ways that were not revealed by our descriptive analysis of 
mobility rates nor accounted for by our weighting strategy for handling missing data. Thus, 
we again encourage caution in interpreting the results for the interim outcomes. Neverthe- 
less, the pattern of results across all outcomes, including those with lower potential attrition 
bias (survey measures of engagement) and those that used the full experimental sample 
(dropout and graduation), is consistent and indicates that Check & Connect did not have a 
measurable positive impact for students in the study. 

A number of noteworthy differences exist between the current study and the previous 
experimental studies of Check & Connect that may account for the difference in findings. 
The first difference is that, unlike the current study, the two previously published studies 
focused on students with learning, emotional, or behavioral disabilities. This difference is 
notable because students receiving special education supports have access to guidance and 
resources that general education students do not (e.g., a case manager, separate from the 
Check & Connect mentor, who monitors their progress, checks in with them regularly, 
assists with modifications and accommodations, and communicates with families; academic 
support within or outside of the general education class; and explicit instruction in study 
skills, social skills, self-management, and decision-making). Check & Connect mentors of 
special education students thus have a network of individuals with whom to connect and 
collaborate regarding the students’ school performance. In contrast, the mentors in the cur- 
rent study found that they did not have an extensive network of resources for the general 
education students they served at the beginning of the study, and they had to work to build 
this network throughout the study. 

A second difference between this study and the two previously published experimental 
studies is the grade level of the students targeted. The earlier studies targeted ninth graders 
who started receiving Check & Connect in Grades 7 and 8 (Sinclair et al., 1998) and ninth 
graders who received Check & Connect from December of their first year of high school 
through four and sometimes five years of intervention (Sinclair et al., 2005). The interven- 
tion for the current study began after students completed ninth grade, mainly because the 
study was designed to test the effect of the program on outcomes through graduation for stu- 
dents identified as at risk based on indicators from Grades 8 and 9. Due to the amount of 
time required to collect and analyze Grade 9 student data, obtain parental consent, and con- 
duct random assignment (approximately three months), the study design did not allow us to 
assign students to mentors until the summer following ninth grade. Moreover, we designed 
the timing of this study to resemble, as closely as possible, the approach that high schools 
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are being encouraged to implement using early warning systems, which is to identify off- 
track students based on early (first-semester) academic behavior and intervene by the latter 
part or at the end of Grade 9 (e.g., Dynarski et al., 2008; Therriault, O;Cummings, Heppen, 
Yerhot, & Scala, 2013). Nevertheless, it is possible that beginning intervention at the end of 
Grade 9 was simply too late, as students were already credit deficient by the time they began 
meeting with their mentor. 

Another important difference between the current study and previous studies of Check & 
Connect concerns the high rate of mobility in our student sample. We found that Check & 
Connect was delivered with fidelity for students who stayed in district schools (with the 
exception of documentation of the check component), but students who did not stay in dis- 
trict high schools generally did not receive Check & Connect as intended. On average, men- 
tors interacted with students in the treatment group who stayed in the district about twice 
per month or more, which is the minimum number of meetings recommended by the pro- 
gram developers. Mentors interacted less frequently with students who left the district 
schools. In addition, mentors were more likely to report check data for nontransfer students, 
and they spent more time per month interacting with nontransfer students relative to trans- 
fer students. The relatively large size of the mentors’ caseloads, as well as the fact that some 
charter schools did not provide mentors with student data or allow mentors to meet with 
students during the school day, contributed to the challenges that student mobility posed for 
the mentors implementing the program. 

The findings of the current study should also be considered in light of other emerging 
investigations of Check & Connect with general education students in different settings. 
These include two randomized trials for which full details are still emerging, but initial 
results suggest mixed findings. Maynard et al. (2014) tested the impact of Check & Connect 
in a Texas district and report small positive effects for middle and high school students on 
academic performance and discipline, but not on attendance. Guryan et al. (2016) examined 
the impact of Check & Connect on students in Grades 1-8 in Chicago and report a positive 
impact on middle school students’ attendance and course performance. However, there were 
no significant positive impacts on test scores and GPA for middle school students, and there 
were no significant impacts on any of the outcomes for elementary students. As these and 
other studies produce full results and details about Check & Connect program implementa- 
tion, setting, and methods, it will be important to synthesize findings across investigations to 
better understand the conditions under which this popular program might have maximal 
positive effects on students. 

The current study findings suggest several considerations for districts implementing 
intensive one-on-one mentoring programs over multiple years with at-risk, highly mobile 
students. First, perhaps districts should consider starting intensive interventions when stu- 
dents enter high school or earlier, before students have had a chance to fall far behind. Sec- 
ond, districts should consider the types of resources and supports that are available for all 
students (i.e., at the universal level, or Tier 1, in response-to-intervention terms) within the 
school or community. In the current study, similar percentages of treatment and control stu- 
dents participated in tutoring, online credit recovery, and college preparatory programs. For 
Check & Connect to be more effective, mentors may need an established network of sup- 
ports that go beyond those that are currently available. Third, districts should carefully con- 
sider caseloads for one-on-one interventions. Particularly for highly mobile students, large 
caseloads may prevent mentors from being able to track down and spend an adequate 
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amount of time working with all of their students. Mentors in this study attempted to con- 
nect with students who left district schools, but they found this process difficult and time 
consuming and were concerned that it was taking time away from other students on their 
caseload. In response, the program developers and implementation team decided during the 
study to prioritize delivery to the nontransfer students. 

These considerations are similar to those noted in a recent systematic review of interven- 
tions aimed at reducing high school dropout rates or increasing school completion rates 
(Freeman & Simonsen, 2015). The authors reviewed a total of 32 peer-reviewed studies that 
used experimental or quasi-experimental designs. Although some interventions had positive 
effects on high school completion rates or negative effects on dropout rates, those that had 
an impact did not share a clearly identifiable set of common features. The authors suggested 
that many types of strategies frequently recommended by theory have not been sufficiently 
tested empirically. These strategies—resembling the considerations we noted above—include 
early intervention, attention to the school organizational structure, integration of reform 
efforts, and attention to the differing needs and responses of different student subgroups. 
Urging the field to acknowledge and confront the complexity of the dropout issue, Freeman 
and Simonsen (2015) argue for the importance of addressing multiple risk factors and the 
need for early intervention, with a corresponding move away from single-component, indi- 
vidual, and small-group interventions. 

In the current study, it is not clear whether putting more emphasis on implementation of 
Check & Connect with transfer students would have yielded different results. In our sample, 
only about half of both treatment and control students remained in district schools during 
the three-year period of implementation. Although Check & Connect might be hypothesized 
to have the largest effect with these more stable students because they had greater access to 
their mentors, our TOT analyses indicate that the program did not have an impact on stu- 
dents who received more of the treatment. But perhaps those more stable students—who 
were less at risk than students who transferred—were less in need of intervention. Perhaps 
to have an overall impact, the program needed to reduce the likelihood of transfer, or find 
more ways to provide services to the transfer students, such as by employing more mentors. 
However, the latter may be simply impractical, particularly if mentors are district employees 
and resources are limited. In general, school district leaders and researchers should place 
more emphasis on how to address student mobility when implementing school-based 
interventions. 

It is also worth noting the distinctions between school-based mentoring and community- 
based mentoring; each may have advantages. School-based mentoring can be integrated into 
the school environment, which aids mentors in advocating on their students’ behalf. Also, 
school-based programs may be better able to reach at-risk youth (Rhodes, 2002), as long as 
students come to school. In contrast, community-based mentoring may allow for mentors 
and mentees to meet more frequently or for longer periods of time, and to focus on broader 
needs of youth, such as exploring job-readiness skills and career opportunities. In addition, 
community-based mentoring may allow more opportunities for mentors and mentees to 
engage in social activities that facilitate relationship building (Langhout, Rhodes, & Osborne, 
2004; Rhodes, 2002). Another advantage of community-based mentoring over school-based 
mentoring is that the relationship does not terminate if the student transfers to another 
school (Rhodes, 2002), a key challenge in the current study. In the current study, Check & 
Connect was mainly implemented as school-based. It is possible that the highly mobile 
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students in the sample could have benefited from a more community-based mentoring 
approach; however, this would have limited mentors’ ability to track students’ academic 
progress and form close ties with school personnel. 

This three-year evaluation of Check & Connect focused on general education students at 
high risk of school dropout. Program delivery began in the summer before tenth grade, 
when students were already significantly behind. As a targeted intervention, the program 
had no observable impact for at-risk general education students on the key outcome meas- 
ures used in the study, which varied from engagement in school to graduation and dropout. 
Future research might combine Check & Connect with universal practices (multicomponent 
efforts) for students at high risk of continued disengagement. Given the recent literature and 
the results of the current study, there is clear urgency to identify and continuously improve 
interventions that engage at-risk students in school and improve the likelihood of their aca- 
demic success. 


Funding 


This research was supported by Grant R305A110252 from the Institute of Education Sciences, U.S. 
Department of Education, to American Institutes for Research. The opinions expressed are those of 
the authors and do not represent the views of the Institute or the U.S. Department of Education. 


ARTICLE HISTORY 


Received 6 July 2016 
Revised 15 February 2017 
Accepted 2 April 2017 


References 


Alliance for Excellent Education. (2011). The high cost of high school dropouts: What the nation pays 
for inadequate high schools (Issue Brief). Washington, DC: Author. Retrieved from http://all4ed. 
org/wp-content/uploads/2013/06/HighCost.pdf 

Appleton, J., Christenson, S. L., Kim, D., & Reschly, A. (2006). Measuring cognitive and psychological 
engagement: Validation of the Student Engagement Instrument. Journal of School Psychology, 44 
(5), 427-445. 

Balfanz, R., & Legters, N. (2006). Closing “dropout factories”: The graduation-rate crisis we know, and 
what can be done about it. Education Week, 25(42), 42-43. 

Biggs, S. A., Musewe, L. O., & Harvey, J. P. (2014). Mentoring and academic performance of black and 
under-resourced urban middle grade students. Negro Educational Review, 65, 64-86. 

Blakeslee, J. E., & Keller, T. E. (2012). Building the youth mentoring knowledge base: Publishing trends 
and coauthorship networks. Journal of Community Psychology, 40(7), 845-859. 

Christenson, S. L., Stout, K. S., & Pohl, A. (2012). Check & Connect: A comprehensive student engage- 
ment intervention: Implementing with fidelity. Minneapolis: University of Minnesota, Institute on 
Community Integration. 

Christenson, S. L., Thurlow, M. L., Sinclair, M. F., Lehr, C. A., Kaibel, C. M., & Reschly, A. L. (2008). 
Check & connect: A comprehensive student engagement intervention manual. Minneapolis, MN: 
University of Minnesota, Institute on Community Integration. 

College Board. (2012). The College Board 2012 National Survey of School Counselors and Administra- 
tors: Report on survey findings: Barriers and supports to school counselor success. Retrieved from 
https://secure-media.collegeboard.org/digitalServices/pdf/nosca/Barriers-Supports_TechReport_Fi 
nal.pdf 


EFFICACY OF CHECK & CONNECT (@) 25 


DuBois, D. L., Portillo, N., Rhodes, J. E., Silverthorn, N., & Valentine, J. C. (2011). How effective are 
mentoring programs for youth? A systematic assessment of the evidence. Psychological Science in 
the Public Interest, 12(2), 57-91. 

Dynarski, M., Clarke, L., Cobb, B., Finn, J., Rumberger, R., & Smink, J. (2008). Dropout preven- 
tion: A practice guide (NCEE 2008-4025). Washington, DC: National Center for Education 
Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of 
Education. 

Dynarski, M., & Wood, R. (1997). Helping high-risk youths: Results from the alternative schools demon- 
stration program. Princeton, NJ: Mathematica Policy Research. 

Finn, J. D. (1989). Withdrawing from school. Review of Educational Research, 59(2), 117-142. 

Freeman, J., & Simonsen, B. (2015). Examining the impact of policy and practice interventions on high 
school dropout and school completion rates: A systematic review of the literature. Review of Educa- 
tional Research, 85(2), 205-248. 

Grannis, J. C. (1994). The dropout prevention initiative in New York City: Educational reforms for at- 
risk students. In R. J. Rossi (Ed.), Schools and students at risk: Context and framework for positive 
change (pp. 182-206). New York, NY: Teachers College Press. 

Greene, J. P., & Winters, M. A. (2005). The effect of residential school choice on public high school grad- 
uation rates (Education Working Paper No. 9.) New York, NY: Manhattan Institute for Policy 
Research. 

Guryan, J., Christenson, S., Claessens, A., Engel, M., Lai, L, Ludwig, J., ... Turner, M. C. (2016). The 
effect of mentoring on school attendance and academic outcomes: A randomized evaluation of the 
Check & Connect program (Working Paper Series WP-16-18). Evanston, IL: Northwestern Institute 
for Policy Research. 

Langhout, R. D., Rhodes, J. E., & Osborne, L. N. (2004). An exploratory study of youth mentoring in 
an urban context: Adolescents’ perceptions of relationship styles. Journal of Youth and Adolescence, 
33(4), 293-306. 

Levin, H., Belfield, C., Muennig, P., & Rouse, C. (2007). The costs and benefits of an excellent education 
for America’s children. New York, NY: Columbia University, Center for Benefit-Cost Studies of 
Education at Teachers College. 

Maynard, B. R., Kjellstrand, E. K., & Thompson, A. M. (2014). Effects of Check & Connect on atten- 
dance, behavior, and academics: A randomized effectiveness trial. Paper presented at the Spring 
Conference of the Society for Research on Educational Effectiveness, March, Washington, DC. 

Rhodes, J. E. (2002). Research corner: School-based mentoring. MENTOR: National Mentoring Part- 
nership. Retrieved from http://www.mentoring.org/downloads/mentoring_1314.pdf 

Rhodes, J. E., & DuBois, D. L. (2008). Mentoring relationships and programs for youth. Current Direc- 
tions in Psychological Science, 17(4), 254-258. 

Ridgeway, G., McCaffrey, D., Morral, A., Burgette, L., Griffin, B. A.. & Burgette, L. (2013). Twang: 
Toolkit for weighting and analysis of nonequivalent groups. R package version 1.311. Retrieved 
from http://CRAN.R-project.org/package=twang 

Sinclair, M. F., Christenson, S. L., Evelo, D. L., & Hurley, C. M. (1998). Dropout prevention for 
youth with disabilities: Efficacy of a sustained school engagement procedure. Exceptional Chil- 
dren, 65(1), 7-21. 

Sinclair, M. F., Christenson, S. L., & Thurlow, M. L. (2005). Promoting school completion of urban 
secondary youth with emotional or behavioral disabilities. Exceptional Children, 71(4), 465-482. 
Snyder, T. D., & Dillow, S. A. (2012). Digest of Education Statistics 2011 (NCES 2012-001). Washing- 
ton, DC: National Center for Education Statistics, Institute of Education Sciences, U.S. Department 

of Education. 

Steinberg, L., (in collaboration with B. Brown & S. Dornbusch). (1996). Beyond the classroom: Why 
school reform has failed and what parents need to do. New York, NY: Simon & Schuster. 

Stetser, M., & Stillwell, R. (2014). Public high school four-year on-time graduation rates and event drop- 
out rates: School years 2010-11 and 2011-12. First look (NCES 2014-391). Washington, DC: U.S. 
Department of Education, National Center for Education Statistics. 


26 J.B. HEPPEN ET AL. 


Therriault, S. B., O;Cummings, M., Heppen, J., Yerhot, L., & Scala, J. (2013). High School Early Warn- 
ing Intervention Monitoring System implementation guide. Washington, DC: American Institutes 
for Research, National High School Center. 

Tierney, J. P., Grossman, J. B., & Resch, N. L. (1995). Making a difference. An impact study of Big 
Brothers/Big Sisters. Public/Private Ventures. Retrieved from http://ppv.issuelab.org/resource/ 
making _a_difference_an_impact_study_of_big_brothers_big_sisters_re_issue_of_1995_study 

U.S. Department of Education, National Center for Education Statistics. (2016). The condition of edu- 
cation 2016 (NCES 2016-144). Washington, DC: Author. 

U.S. Department of Education, Office of Special Education and Rehabilitative Services, Office of Spe- 
cial Education Programs. (2006). 26th annual (2004) report to Congress on the implementation of 
the Individuals with Disabilities Education Act, Vol. 1. Washington, DC: Author. 

What Works Clearinghouse. (2015). Check & Connect dropout prevention (WWC Intervention 
Report). Washington, DC: U.S. Department of Education, Institute of Education Sciences. 

Wooldridge, J. M. (2007). Inverse probability weighted estimation for general missing data problems. 
Journal of Econometrics, 141(2), 1281-1301. 


EFFICACY OF CHECK & CONNECT (@) 27 


